feat(promptfoo): validate exported provider configs#1709
Merged
Conversation
Deploying agentv with
|
| Latest commit: |
947489b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://33340354.agentv.pages.dev |
| Branch Preview URL: | https://feat-av-lbcv-promptfoo-expor.agentv.pages.dev |
1438ab3 to
7b74d65
Compare
Base automatically changed from
feat/av-uttb-provider-surface-migration-docs
to
main
July 7, 2026 02:30
639d396 to
947489b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/export-promptfoo-config.tspath that exports representative AgentV eval YAML into a Promptfoo config for validation.agentv:codex-clito a generated Promptfoo-loadable TypeScript custom provider module, and keep validation free of live provider calls.environmentsubset into a generated PromptfoobeforeAllextension, while passing the resolved isolated workdir through provider config, default-test vars, and metadata.environmentexport with a clear diagnostic instead of degrading isolation, image/context, mounts, services, resources, secrets, or provenance.Promptfoo Exportjob that runs exporter tests, exports the host environment fixture, and validates it withpromptfoo validate config.Dependency Notes
This PR is stacked on
feat/av-uttb-provider-surface-migration-docsbecause it builds on that provider-surface docs/migration branch. I did not mergefeat/av-ctfu-providers-schema: that active worktree currently has uncommitted parser/schema changes owned by workerav-ctfu-providers-schema, so this exporter stays out of those core parser files.Promptfoo
0.121.15and0.121.17both reject provider IDs with afile://...ts:callApisuffix during config validation. The exporter therefore emits the accepted TypeScript custom provider module formfile://.agentv/generated/promptfoo/providers/codex-cli-provider.tswhile the generated module also exposescallApiinternally.Validation
bun run validate:promptfoo-exportbunx biome check scripts/export-promptfoo-config.ts scripts/export-promptfoo-config.test.ts scripts/fixtures/promptfoo-export/host-environment.agentv.yaml scripts/fixtures/promptfoo-export/docker-environment-unsupported.agentv.yaml .github/workflows/ci.yml apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdxtmpdir=$(mktemp -d) && bun scripts/export-promptfoo-config.ts --input scripts/fixtures/promptfoo-export/host-environment.agentv.yaml --output "$tmpdir/promptfooconfig.yaml" && bunx promptfoo@0.121.15 validate config -c "$tmpdir/promptfooconfig.yaml"bun run typecheckgit diff --checkLive dogfood was not run because this change only exports and validates/config-loads Promptfoo config and does not execute AgentV eval/provider/grader behavior.